Review: Hypothesis Testing

Review: Hypothesis Testing

Review: Hypothesis Testing

Review: Hypothesis Testing

Review: Hypothesis Testing

Review: Hypothesis Testing

Review: Hypothesis Testing

Consider Hypothetical RCT

Inference

For any inference procedure:

• Formally state null and alternative hypotheses,

• Choose a test statistic to measure evidence against the null hypothesis,

• Calculate distribution of test statistic under the null to construct p-values, perform hypothesis test.

Null: No Treatment Effect on Average

• Null of no effect on average,
  versus alternative of some effect on average.

• Commonly tested with t-test, based on asymptotic normality.

Null: No Treatment Effect on Anyone

• Null of no effect on any subject,
  versus alternative of some effect on someone.

• Randomization inference will test strong null as we can find distribution of test statistic under sharp null exploiting known randomization procedure.

Consider Hypothetical RCT

Consider Hypothetical RCT

Consider Hypothetical RCT

Randomization Inference:

• Sharp null,
  • \(\mathbb{H}_0: Y_{1,i} - Y_{0,i}=0\) for all \(i\), vs.
  • \(\mathbb{H}_1: Y_{1,i} - Y_{0,i} \ne 0\) for some \(i\).

Distribution of \(T_N\) Under Sharp Null

• There are \({5 \choose 3}=10\) ways to assign treatment to \(3\) out of \(5\) subjects.

• Under sharp null, we can calculate value of test statistic for each way to assign treatment.

• Suppose randomization procedure puts equal probability on each possible way to assign treatment. Then we can find distribution of test statistic under sharp null.

Distribution of \(T_N\) Under Sharp Null

Under sharp null, we know value of \(T_N\) for each possible treatment assignment. Using that randomization procedure makes each possible treatment assignment equally likely, we thus know distribution of \(T_N\) under the sharp null.

Distribution of \(T_N\) Under Sharp Null

Under sharp null:

Randomization Inference: PROGRESA

Complication:

• Randomization done at village level, not child level.
  • Modify procedure to consider possible treatment as village level.

Randomization Inference: PROGRESA

Complication:

• Randomization of 308 out of 491 villages.
  • Very large number of ways to assign treatment to 308 out of 491 villages. . .

choose(491,308)

[1] 2.452172e+139

Randomization Inference: PROGRESA

Complication:

• Not feasible to enumerate all possible \({491 \choose 308}\) ways to assign treatment.

Randomization Inference: PROGRESA

Randomization Inference: PROGRESA

Randomization Inference: PROGRESA

                   term   estimate two_tailed_p_value
1 Custom Test Statistic 0.03923947              6e-04

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Random assignment procedure: Cluster random assignment 
Number of units: 14996 
Number of clusters: 491
Number of treatment arms: 2 
The possible treatment categories are 0 and 1.
The number of possible random assignments is approximately infinite. 
The probabilities of assignment are constant across units: 
   prob_0    prob_1 
0.3727088 0.6272912 

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

Code:

library(ri2, quietly = TRUE)

#declaring randomization procedure
n.0<- length(unique(dfPost$sooloca))   
m.0<- length(unique(subset(dfPost,
          dfPost$treat==1)$sooloca))  
declaration_ra<-declare_ra(N=n.0,
          m=m.0,clusters=dfPost$sooloca)
declaration_ra

# perform randomization inference 
ri_out <-  conduct_ri(
    test_function = function(d){
       abs(mean(d[d$treat==1,]$school) 
         -mean(d[d$treat==0,]$school))}, 
    declaration = declaration_ra, 
    assignment = "treat", 
    outcome = "school", 
    data = dfPost, 
    sims = 10000 
  )

# showing results
plot(ri_out)
ri_out

                   term   estimate two_tailed_p_value
1 Custom Test Statistic 0.03923947              6e-04

Randomization Inference: Subgroups

• For any subgroup defined at baseline, we can test sharp null of no effect within that subgroup.
  • For example, for boys and girls separately
  • For example, by highest grade completed at baseline.

Randomization Inference: Boys

Randomization Inference: Girls

Multiple Hypothesis Testing

• Suppose our goal is to determine for which (if any) of the grades does the treatment have an effect, i.e., we want to test sharp null for each of the 10 grades.

  • Consider null hypothesis and alternative for each \(j=1,...,10\),
    • \(\mathbb{H}_{0,j}\): sharp null holds for grade \(j\),
      vs
    • \(\mathbb{H}_{1,j}\) : sharp null does not hold for grade \(j\).

Multiple Hypothesis Testing

• Consider null hypothesis and alternative for each \(j=1,...,10\),
  • \(\mathbb{H}_{0,j}\): sharp null holds for grade \(j\),
    vs
  • \(\mathbb{H}_{1,j}\) : sharp null does not hold for grade \(j\).
• We are thus testing a family of \(J\) null hypotheses,
  \(\mathbb{H}_{0,1}\),…,\(H_{0,J}\). Testing a family of \(J \ge 2\) null hypothesis is called a multiple hypothesis testing problem.

FWER

• In our PROGRESA example, the FWER is the probability of falsely concluding that the treatment has an effect for at least one grade for which it actually has no effect.

FWER

• If we test each of \(J \ge 2\) null hypotheses separately, each at the \(\alpha\) significance level, the FWER will often be far higher than \(\alpha\), especially if the number of true null hypotheses is large.

Correcting p-values to Control FWER

• Instead of testing so that the probability of falsely rejecting each null is at most \(\alpha\), we we can design our testing procedure so that the FWER is at most \(\alpha\).

• Let \(\widehat p_j\) denote the p-value for testing \(\mathbb{H}_{0,j}\). Multiple hypothesis testing procedures typically use as inputs \((\widehat p_{1},...,\widehat p_{J})\), and “correct” the p-values for the multiplicity of hypotheses.

• The simplest such procedure is the Bonferonni Procedure.

Bonferonni

• \(J \times \widehat p_j\) is p-value corrected for multiple hypothesis testing.

• Can implement in R using function p.adjust with option method="bonferroni".

  • For example, if p is a vector of p-values, then p.adjust(p, method="bonferroni") returns vector of p-values with Bonferroni correction.

Bonferonni

• Advantage:
  • Controls FWER.
• Disadvantage:
  • Correction makes it far harder to reject null hypotheses, increasing probability of Type II errors for false null hypotheses.